Kusama state version switch and migration. #7015

cheme · 2023-04-06T09:04:35Z

This PR adds state-migration pallet to kusuma to allow switch to trie state version 1.
It should use the same configuration as the ones previously used in westend (#6336)

It may be pertinent to avoid adding this in a runtime upgrade that already have others migration.

bkchr

So this will enable automatic migration of Kusama after the runtime upgrade is applied?

bkchr · 2023-04-09T23:13:33Z

runtime/kusama/src/lib.rs

+	use sp_std::prelude::*;
+
+	/// Initialize an automatic migration process.
+	pub struct InitMigrate;


You need to add it to the list of migrations.

xlc · 2023-04-09T23:19:14Z

Do we have confirmation this is fully compatible with all the ecosystem tooling? e.g smoldot and alternative polkadot clients

bkchr · 2023-04-09T23:51:49Z

Everything that can run Rococo/Westend is already compatible with this change as state version V1 is already running there. Smoldot is also ready AFAIK.

cheme · 2023-04-10T08:02:28Z

So this will enable automatic migration of Kusama after the runtime upgrade is applied?

Yes without privileged account (could be an idea to be able to emergency switch off but that would mean giving a specific power to some account/owner, a bit awkward in my opinion).

bkchr · 2023-04-10T10:17:31Z

In what case would you think that we need an emergency stop?

cheme · 2023-04-10T10:36:44Z

In what case would you think that we need an emergency stop?

Bug, but not much value in it (as can only stop if the blocks are still processing).

bkchr · 2023-05-10T15:41:14Z

@cheme please merge master.

kianenigma

No major issues, but should fix the comment prior to merge.

kianenigma · 2023-05-10T17:28:02Z

runtime/kusama/src/lib.rs

+	type SignedFilter = frame_support::traits::NeverEnsureOrigin<AccountId>;
+
+	// Use same weights as substrate ones.
+	type WeightInfo = pallet_state_trie_migration::weights::SubstrateWeight<Runtime>;


Would be more appropriate to benchmark it properly in this runtime.

yes, not sure how much change it will be to do so (in theory there is no reason to see significant change: only wasm bin size may interfere a tiny bit maybe).

kianenigma · 2023-05-10T17:28:18Z

runtime/kusama/src/lib.rs

+	type Currency = Balances;
+	type SignedDepositPerItem = MigrationSignedDepositPerItem;
+	type SignedDepositBase = MigrationSignedDepositBase;
+	type ControlOrigin = EnsureRoot<AccountId>;


Sure? it would be hard to acquire.

I am not very comfortable with having a privilege account.
Actually I recently reconsider bringing back the inherent version of the migration (the main issue was the code was a bit everywhere and less cleanly separated, but looking backward, it would have really feel more comfortable to me).

kianenigma · 2023-05-10T17:29:10Z

runtime/kusama/src/lib.rs

+			if MigrationProcess::<Runtime>::get() == Default::default() &&
+				AutoLimits::<Runtime>::get().is_none()
+			{
+				AutoLimits::<Runtime>::put(Some(MigrationLimits { item: 160, size: 204800 }));


Some comment about how this number is chosen would be good.

chevdor · 2023-05-12T07:17:43Z

runtime/kusama/src/lib.rs

+	migrations::V0941,
+	migrations::V0942,
+	migrations::Unreleased,
+	init_state_migration::InitMigrate,


Is there a reason why not to move that into migrations::Unreleased,?

migrations::Unreleased migrations do run, we just don't know yet in which of the upcoming Version it will be included.

At release time, merged Unreleased migrations will moved to a new VXXXX migration set.

👍 I will do, I just merged to master a bit too fast.
(currently I am trying to get better limit but encounter some limitation with try runtime, but will certainly get new limits soon).

chevdor · 2023-05-12T09:58:55Z

runtime/kusama/src/lib.rs

+pub type Migrations =
+		(migrations::V0940, migrations::V0941,
+	migrations::V0942, migrations::Unreleased);


I am no fan of what rustfmt does here.
The list you had before is much more readable.

I would suggest adding #[rustfmt_skip] for this Migrations type as well as for Unreleased and keep one migration / version per line. That also helps with your comment about having this single migration in a release.

chevdor · 2023-05-12T10:00:45Z

runtime/kusama/src/lib.rs

+	pub type Unreleased = (init_state_migration::InitMigrate,);
 }


Your comment:

It may be pertinent to avoid adding this in a runtime upgrade that already have others migration.

would be suited here so whoever does the move from Unreleased to VNext is aware of it.
We can then drop the comment.

cheme · 2023-05-15T16:46:05Z

@bkchr @kianenigma I did change the limits (see https://forum.polkadot.network/t/state-trie-migration/852/11).
Does the weight column looks fine (only in the targetted tab, I am not really use to weight so would appreciate feedback on the value (value is from on_init only).

bkchr · 2023-05-15T21:55:33Z

runtime/kusama/src/lib.rs

@@ -2405,7 +2405,11 @@ mod init_state_migration {
 			if MigrationProcess::<Runtime>::get() == Default::default() &&
 				AutoLimits::<Runtime>::get().is_none()
 			{
-				AutoLimits::<Runtime>::put(Some(MigrationLimits { item: 160, size: 204800 }));
+				// We use limits to target 600ko proofs per block and
+				// avg 800_000_000_000 of weight per block.


800 seconds?

800_000_000_000 is 800ms I believe

Can lower to 600 (using 3200 instead of 4800 move to 510 blocks the migration time so not taking really longer).

Could make sense to calculate better weight as I believe it is an overestimate (iterating on data since iterator improvement may be cheaper than random data access).

800ms sounds good! For some reason I thought we measure ref time in nanoseconds, but ty @crystalin

bkchr · 2023-07-04T14:56:46Z

@cheme what is the status here?

cheme · 2023-07-04T15:02:00Z

Forgot to merge the PR :( , it was totally ready iirc, will rebase and merge.

cheme · 2023-07-04T15:07:12Z

bot merge

paritytech-processbot · 2023-07-04T15:07:19Z

Waiting for commit status.

runtime/kusama/src/lib.rs

paritytech-processbot · 2023-07-04T15:17:32Z

Merge cancelled due to error. Error: Statuses failed for 9f4c073

cheme · 2023-07-04T15:56:55Z

bot merge

paritytech-processbot · 2023-07-04T15:57:03Z

Error: Statuses failed for 18cc436

cheme · 2023-07-04T16:17:13Z

bot merge

cheme · 2023-07-04T16:19:42Z

ok, migration is pending in unreleased. @chevdor is it enought to get included in next runtime build?

chevdor · 2023-07-04T16:30:55Z

ok, migration is pending in unreleased. @chevdor is it enought to get included in next runtime build?

yes

cheme · 2023-07-04T16:59:50Z

ok, migration is pending in unreleased. @chevdor is it enought to get included in next runtime build?

yes

Ok, if for any reason it ends up not being included in next runtime, then it would be better if:

polkadot/runtime/kusama/src/lib.rs

Line 143 in 032d3eb

state_version: 1,

stays at 0 (it breaks warpsync).

Polkadot-Forum · 2023-07-06T07:22:02Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/state-trie-migration/852/13

State version switch and migration init

b790797

paritytech-ci requested review from a team April 6, 2023 09:05

cheme added E1-database_migration PR introduces code that does a one-way migration of the database. T0-node This PR/Issue is related to the topic “node”. T1-runtime This PR/Issue is related to the topic “runtime”. B1-note_worthy Changes should be noted in the release notes labels Apr 6, 2023

fix

4a8a379

bkchr approved these changes Apr 9, 2023

View reviewed changes

paritytech-ci requested a review from a team April 9, 2023 23:14

migration declaration (last position, westend was first)

dcdefbe

cheme added 2 commits May 10, 2023 17:46

Merge branch 'master' into kusuma-migration2

a69b387

revert unrelated cargo fmt

707b986

kianenigma approved these changes May 10, 2023

View reviewed changes

chevdor reviewed May 12, 2023

View reviewed changes

move to unrelease

f4bc170

cheme removed the T0-node This PR/Issue is related to the topic “node”. label May 12, 2023

cheme added 2 commits May 12, 2023 11:49

fmt?

925074a

fmt

efa685a

cheme changed the title ~~Kusuma state version switch and migration.~~ Kusama state version switch and migration. May 12, 2023

chevdor reviewed May 12, 2023

View reviewed changes

Change size of processing per blocks

bdd9ae3

bkchr reviewed May 15, 2023

View reviewed changes

Merge branch 'master' into kusuma-migration2

9f4c073

liamaharon reviewed Jul 4, 2023

View reviewed changes

runtime/kusama/src/lib.rs Show resolved Hide resolved

Dispatch error

18cc436

paritytech-processbot bot merged commit 032d3eb into paritytech:master Jul 4, 2023

crystalin mentioned this pull request Oct 20, 2023

Update substrate/polkadot/cumulus from v0.9.43 to v1.1.0 moonbeam-foundation/moonbeam#2535

Closed

JesseAbram mentioned this pull request Oct 26, 2023

Update substrate and subxt entropyxyz/entropy-core#435

Merged

zjb0807 mentioned this pull request May 10, 2024

update state version to 1 on mandala AcalaNetwork/Acala#2759

Merged

		pub type Unreleased = (init_state_migration::InitMigrate,);
		}

Kusama state version switch and migration. #7015

Kusama state version switch and migration. #7015

Conversation

cheme commented Apr 6, 2023 • edited Loading

bkchr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xlc commented Apr 9, 2023

bkchr commented Apr 9, 2023

cheme commented Apr 10, 2023

bkchr commented Apr 10, 2023

cheme commented Apr 10, 2023

bkchr commented May 10, 2023

kianenigma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheme commented May 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkchr commented Jul 4, 2023

cheme commented Jul 4, 2023

cheme commented Jul 4, 2023

paritytech-processbot bot commented Jul 4, 2023

paritytech-processbot bot commented Jul 4, 2023

cheme commented Jul 4, 2023

paritytech-processbot bot commented Jul 4, 2023

cheme commented Jul 4, 2023

cheme commented Jul 4, 2023

chevdor commented Jul 4, 2023

cheme commented Jul 4, 2023

Polkadot-Forum commented Jul 6, 2023

cheme commented Apr 6, 2023 •

edited

Loading